Allow multipart downloading when pulling from remote - #346

Eric,

Thanks for putting in the effort. As an illustration, this did its job: it helped clarify the direction I’ll take. I don’t yet know the timeline.

The next step will be per-request selection via api.GetArgs.

Most of the mechanism already exists: apc.HdrBlobDownload, apc.HdrBlobWorkers, and apc.HdrBlobChunk. To keep the development incremental and consistent with the existing code, I may expose those headers directly through GetArgs, or add dedicated typed fields. That remains to be seen.

The advantage is that this can be one self-contained commit, with no additional persistent configuration to validate, propagate, and maintain. The tradeoff is that the caller must know it is about to read a large object.

A subsequent step may add a server-side default: cluster config would provide the default, while bucket properties would inherit and optionally override it.

One additional motivation for that second step is support for third-party clients such as Boto3/Botocore.

The threshold decision belongs at the bucket level. A bucket containing 20GB shards and one containing 100KB JSON objects want opposite defaults, and a single cluster-wide setting cannot express that.

Again, I appreciate the push on this one!

PS. Speaking of config, we actually already have it. But let us do part-one first, and then due diligence on the second one.

Some questions:

The next step will be per-request selection via api.GetArgs.

What's still needed here? I think the existing headers already expose the things that we want?

PS. Speaking of config, we actually already have it

Do you have a pointer to it? I didn't see anything like this when I looked at it

Do you have a pointer to it?

https://github.com/NVIDIA/aistore/blob/main/cmn/api.go#L68

and further:

$ git grep "Bprops.*Chunks" ais/t*
ais/tgtobj.go:252:      maxMonoSize := int64(poi.lom.Bprops().Chunks.MaxMonolithicSize)
ais/tgtobj.go:259:              return poi.chunk(int64(poi.lom.Bprops().Chunks.ChunkSize))
ais/tgtobj.go:867:      case goi.lom.Bprops().Chunks.AutoEnabled():
ais/tgtobj.go:1808:     dstMaxMonoSize := dst.Bprops().Chunks.MaxMonolithicSize
ais/tgtobj.go:1831:     case lom.Bprops().Chunks.MaxMonolithicSize != dstMaxMonoSize && lom.Lsize() > int64(dstMaxMonoSize):
ais/tgtobj.go:1833:             res = coi._chunk(t, lom, dst, int64(dst.Bprops().Chunks.ChunkSize))
$ 

is it really desirable to couple multipart download config to chunking config for uploads? Intuitively I would think people may want different configurations for these.

The next step will be per-request selection via api.GetArgs.
What's still needed here? I think the existing headers already expose the things that we want?

bump on this question

I'm also happy to implement this if that's helpful - I'd like to get this resolved in a somewhat timely fashion

Hi @eric-higgins-ai, feel free to proceed with part one. Please keep it focused on the Go client API - the server path already exists. Add an optional typed blob-download setting to api.GetArgs: non-nil enables blob download.
Translate those values into the existing HdrBlobDownload, HdrBlobChunk, and HdrBlobWorkers headers in the shared GetArgs request-building path.
Make sure it handles potential conflicts with values already supplied through GetArgs.Header. Add focused tests for defaults, explicit values, and conflicts. Once it’s ready, I’ll review it.

@Nahemah1022 it's still not clear to me why we're doing this - why do y'all want HdrBlobDownload, HdrBlobChunk, and HdrBlobWorkers to be specified via query params (?) instead of headers? It doesn't seem really related to my request, which was to configure these headers on the server side

@eric-higgins-ai We’re splitting this into two steps. What I mentioned previously is the first step as a client-controlled BlobThreshold in GetArgs. I’ve implemented it in commit 5631038

The follow-up is a server-side bucket property that automatically uses blob download for cold GET objects above a configured threshold, without client arguments. That aligns with your PR’s goal, but has additional caveats, so we’ll handle it soon and keep this thread updated.

Hi @Nahemah1022, any update on the server-side follow-up (bucket property that auto-routes cold GETs above a threshold through blob download)?

@wendy-ouyang-ai Thank you for your patience and sorry for the delayed update. We were wrapped up with high-priority items for the previous 5.0 release. We've resumed work on this topic now. The goal is to add a bucket property that automatically routes coldGet through the blob download path when objects exceed a configured size threshold. This feature is planned for the upcoming 5.1 minor release shortly.

@wendy-ouyang-ai Thank you for your patience and sorry for the delayed update. We were wrapped up with high-priority items for the previous 5.0 release. We've resumed work on this topic now. The goal is to add a bucket property that automatically routes coldGet through the blob download path when objects exceed a configured size threshold. This feature is planned for the upcoming 5.1 minor release shortly.

Thanks! Do you have a rough ETA? Once the change lands on main, we can cherry-pick the commit onto our deployment without waiting for the full 5.1 release.

@wendy-ouyang-ai We're aiming to land the initial implementation on main within the next few days. Note that this feature is planned in two phases:

• Serial mode: The blob-download job persists the entire object and its chunks in AIS first before responding to the client. The client waits for the full download to finish before receiving the first byte. This phase is straightforward and should be ready in the next couple of days.
• Streaming mode: AIS streams individual chunk data back to the client as soon as they land in the cluster. Because this requires additional memory management overhead, it will take a bit longer to complete.

If serial mode fits your workflow, you should be able to cherry-pick the initial commit soon!

@wendy-ouyang-ai We're aiming to land the initial implementation on main within the next few days. Note that this feature is planned in two phases:

• Serial mode: The blob-download job persists the entire object and its chunks in AIS first before responding to the client. The client waits for the full download to finish before receiving the first byte. This phase is straightforward and should be ready in the next couple of days.
• Streaming mode: AIS streams individual chunk data back to the client as soon as they land in the cluster. Because this requires additional memory management overhead, it will take a bit longer to complete.

If serial mode fits your workflow, you should be able to cherry-pick the initial commit soon!

Serial mode works for us. We don't care too much about first-byte arrival time. Please let us know once the commit is ready to cherry-pick. Thanks @Nahemah1022 !

@wendy-ouyang-ai Thanks for the patience! We're actively working through the implementation now and expect to have it landed shortly, ideally by the end of this week. I'll drop an update here as soon as the PR is merged.